Tensor Principal Component Analysis Research Articles

Statistical-computational trade-offs in tensor PCA and related problems via communication complexity

Statistical-computational trade-offs in tensor PCA and related problems via communication complexity

Optimized Tensor Decomposition and Principal Component Analysis Outperforming State-of-the-Art Methods When Analyzing Histone Modification Chromatin Immunoprecipitation Profiles

It is difficult to identify histone modification from datasets that contain high-throughput sequencing data. Although multiple methods have been developed to identify histone modification, most of these methods are not specific to histone modification but are general methods that aim to identify protein binding to the genome. In this study, tensor decomposition (TD) and principal component analysis (PCA)-based unsupervised feature extraction with optimized standard deviation were successfully applied to gene expression and DNA methylation. The proposed method was used to identify histone modification. Histone modification along the genome is binned within the region of length L. Considering principal components (PCs) or singular value vectors (SVVs) that PCA or TD attributes to samples, we can select PCs or SVVs attributed to regions. The selected PCs and SVVs further attribute p-values to regions, and adjusted p-values are used to select regions. The proposed method identified various histone modifications successfully and outperformed various state-of-the-art methods. This method is expected to serve as a de facto standard method to identify histone modification. For reproducibility and to ensure the systematic analysis of our study is applicable to datasets from different gene expression experiments, we have made our tools publicly available for download from gitHub.

Selective multiple power iteration: from tensor PCA to gradient-based exploration of landscapes

Selective multiple power iteration: from tensor PCA to gradient-based exploration of landscapes

Spectral Methods from Tensor Networks

A tensor network is a diagram that specifies a way to “multiply” a collection of tensors together to produce another tensor (or matrix). Many existing algorithms for tensor problems (such as tensor decomposition and tensor PCA), although they are not presented this way, can be viewed as spectral methods on matrices built from simple tensor networks. In this work we leverage the full power of this abstraction to design new algorithms for certain continuous tensor decomposition problems. An important and challenging family of tensor problems comes from orbit recovery, a class of inference problems involving group actions (inspired by applications such as cryo-electron microscopy). Orbit recovery problems over finite groups can often be solved via standard tensor methods. However, for infinite groups, no general algorithms are known. We give a new spectral algorithm based on tensor networks for one such problem: continuous multi-reference alignment over the infinite group SO(2). Our algorithm extends to the more general heterogeneous case.

Tensor Principal Component Analysis in High Dimensional CP Models

The CP decomposition for high dimensional non-orthogonal spiked tensors is an important problem with broad applications across many disciplines. However, previous works with theoretical guarantee typically assume restrictive incoherence conditions on the basis vectors for the CP components. In this paper, we propose new computationally efficient composite PCA and concurrent orthogonalization algorithms for tensor CP decomposition with theoretical guarantees under mild incoherence conditions. The composite PCA applies the principal component or singular value decompositions twice, first to a matrix unfolding of the tensor data to obtain singular vectors and then to the matrix folding of the singular vectors obtained in the first step. It can be used as an initialization for any iterative optimization schemes for the tensor CP decomposition. The concurrent orthogonalization algorithm iteratively estimates the basis vector in each mode of the tensor by simultaneously applying projections to the orthogonal complements of the spaces generated by other CP components in other modes. It is designed to improve the alternating least squares estimator and other forms of the high order orthogonal iteration for tensors with low or moderately high CP ranks, and it is guaranteed to have second or higher order convergence when the error of any given initial estimator is bounded by a small constant. Our theoretical investigation provides estimation accuracy and convergence rates for the two proposed algorithms. Both proposed algorithms are applicable to deterministic tensor, its noisy version, and the order- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2K$ </tex-math></inline-formula> covariance tensor of order- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> tensor data in a factor model with uncorrelated factors. Simulation experiments demonstrate significant practical superiority of our approach over existing methods.

Low-Rank Tensor Subspace Decomposition With Weighted Group Sparsity for the Acceleration of Non-Cartesian Dynamic MRI.

Dynamic MR imaging often requires long scan time, and acceleration of data acquisition is highly desirable in clinical applications. We proposed a Low-rank Tensor subspace decomposition with Weighted Group Sparsity (LTWGS) algorithm for non-Cartesian dynamic MRI. The proposed algorithm introduces the weighted group sparse constraints together with the subspace decomposition technique into the framework of low-rank tensor and sparse decomposition to better utilize the sparsity in the data. LTWGS increases the PSNR values by 1.97 dB, 2.03 dB, and 2.83 dB compared with PROST (patch-based reconstruction), SRTPCA (smooth robust tensor principal component analysis), and LRTES (low-rank tensor with "explicit subspace") in the dynamic abdominal imaging at an acceleration rate R = 25. LTWGS increases the PSNR values by 2.42 dB and 3.57 dB compared with PROST and LRTES in DCE liver imaging at R = 25. LTWGS increases the PSNR values by 1.40 dB and 1.96 dB compared with PROST and SRTPCA in cardiac cine imaging at R = 25. Jointly using group sparsity and sparsity can obtain better results than that using group sparsity alone, and weighted regularization can achieve better results than that without weighted regularization. The proposed algorithm results in reduced reconstruction error and improved image structural similarity in comparison with several state-of-the-art methods at relatively high acceleration factors. The proposed algorithm has the potential in various dynamic MRI application scenarios.

Robust Tensor Decomposition Based Background/Foreground Separation in Noisy Videos and Its Applications in Additive Manufacturing

Background/foreground separation is one of the most fundamental tasks in computer vision, especially for video data. Robust PCA (RPCA) and its tensor extension, namely, Robust Tensor PCA (RTPCA), provide an effective framework for background/foreground separation by decomposing the data into low-rank and sparse components, which contain the background and the foreground (moving objects), respectively. However, in real-world applications, the video data is contaminated with noise. For example, in metal additive manufacturing (AM), the processed X-ray video to study melt pool dynamics is very noisy. RPCA and RTPCA are not able to separate the background, foreground, and noise simultaneously. As a result, the noise will contaminate the background or the foreground or both. There is a need to remove the noise from the background and foreground. To achieve the three components decomposition, a smooth sparse Robust Tensor Decomposition (SS-RTD) model is proposed to decompose the data into static background, smooth foreground, and noise, respectively. Specifically, the static background is modeled by the low-rank tucker decomposition, the smooth foreground (moving objects) is modeled by the spatio-temporal continuity, which is enforced by the total variation regularization, and the noise is modeled by the sparsity, which is enforced by the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell _{1}$ </tex-math></inline-formula> norm. An efficient algorithm based on alternating direction method of multipliers (ADMM) is implemented to solve the proposed model. Extensive experiments on both simulated and real data demonstrate that the proposed method significantly outperforms the state-of-the-art approaches for background/foreground separation in noisy cases. Note to Practitioners—This work is motivated by melt pool detection in metal additive manufacturing where the processed X-ray video from the monitoring system is very noisy. The objective is to recover the background with porosity defects and the foreground with melt pool in the presence of noise. Existing methods fail to separate the noise from the background and foreground since RPCA and RTPCA have only two components, which cannot explain the three components in the data. This paper puts forward a smooth sparse Robust Tensor Decomposition by decomposing the tensor data into low-rank, smooth, and sparse components, respectively. It is a highly effective method for background/foreground separation in noisy case. In the case studies on simulated video and X-ray data, the proposed method can handle non-additive noise, and even the case of high noise-ratio. In the proposed algorithm, there is only one tuning parameter <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\lambda $ </tex-math></inline-formula> . Based on the case studies, our method achieves satisfying performance by taking any <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\lambda \in [{0.2,1}]$ </tex-math></inline-formula> with anisotropic total variation regularization. With this observation, practitioners can apply the proposed method without extensive parameter tuning work. Furthermore, the proposed method is also applicable to other popular industrial applications. Practitioners can use the proposed SS-RTD for degradation processes monitoring, where the degradation image contains the static background, anomaly, and random disturbance, respectively.

Adapted tensor decomposition and PCA based unsupervised feature extraction select more biologically reasonable differentially expressed genes than conventional methods

Tensor decomposition- and principal component analysis-based unsupervised feature extraction were proposed almost 5 and 10 years ago, respectively; although these methods have been successfully applied to a wide range of genome analyses, including drug repositioning, biomarker identification, and disease-causing genes’ identification, some fundamental problems have been identified: the number of genes identified was too small to assume that there were no false negatives, and the histogram of P values derived was not fully coincident with the null hypothesis that principal component and singular value vectors follow the Gaussian distribution. Optimizing the standard deviation such that the histogram of P values is as much as possible coincident with the null hypothesis results in an increase in the number and biological reliability of the selected genes. Our contribution was that we improved these methods so as to be able to select biologically more reasonable differentially expressed genes than the state of art methods that must empirically assume negative binomial distributions and dispersion relation, which is required for the selecting more expressed genes than less expressed ones, which can be achieved by the proposed methods that do not have to assume these.

Robust Tensor SVD and Recovery With Rank Estimation.

Tensor singular value decomposition (t-SVD) has recently become increasingly popular for tensor recovery under partial and/or corrupted observations. However, the existing t -SVD-based methods neither make use of a rank prior nor provide an accurate rank estimation (RE), which would limit their recovery performance. From the practical perspective, the tensor RE problem is nontrivial and difficult to solve. In this article, we, therefore, aim to determine the correct rank of an intrinsic low-rank tensor from corrupted observations based on t-SVD and further improve recovery results with the estimated rank. Specifically, we first induce the equivalence of the tensor nuclear norm (TNN) of a tensor and its f -diagonal tensor. We then simultaneously minimize the reconstruction error and TNN of the f -diagonal tensor, leading to RE. Subsequently, we relax our model by removing the TNN regularizer to improve the recovery performance. Furthermore, we consider more general cases in the presence of missing data and/or gross corruptions by proposing robust tensor principal component analysis and robust tensor completion with RE. The robust methods can achieve successful recovery by refining the models with correct estimated ranks. Experimental results show that the proposed methods outperform the state-of-the-art methods with significant improvements.

Projection in genomic analysis: A theoretical basis to rationalize tensor decomposition and principal component analysis as feature selection tools.

Identifying differentially expressed genes is difficult because of the small number of available samples compared with the large number of genes. Conventional gene selection methods employing statistical tests have the critical problem of heavy dependence of P-values on sample size. Although the recently proposed principal component analysis (PCA) and tensor decomposition (TD)-based unsupervised feature extraction (FE) has often outperformed these statistical test-based methods, the reason why they worked so well is unclear. In this study, we aim to understand this reason in the context of projection pursuit (PP) that was proposed a long time ago to solve the problem of dimensions; we can relate the space spanned by singular value vectors with that spanned by the optimal cluster centroids obtained from K-means. Thus, the success of PCA- and TD-based unsupervised FE can be understood by this equivalence. In addition to this, empirical threshold adjusted P-values of 0.01 assuming the null hypothesis that singular value vectors attributed to genes obey the Gaussian distribution empirically corresponds to threshold-adjusted P-values of 0.1 when the null distribution is generated by gene order shuffling. For this purpose, we newly applied PP to the three data sets to which PCA and TD based unsupervised FE were previously applied; these data sets treated two topics, biomarker identification for kidney cancers (the first two) and the drug discovery for COVID-19 (the thrid one). Then we found the coincidence between PP and PCA or TD based unsupervised FE is pretty well. Shuffling procedures described above are also successfully applied to these three data sets. These findings thus rationalize the success of PCA- and TD-based unsupervised FE for the first time.

High-order manifold regularized multi-view subspace clustering with robust affinity matrices and weighted TNN

Multi-view subspace clustering achieves impressive performance for high-dimensional data. However, many of these models do not sufficiently mine the intrinsic information among samples and consider the robustness problem of the affinity matrices, resulting in the degradation of clustering performance. To address these problems, we propose a novel high-order manifold regularized multi-view subspace clustering with robust affinity matrices and a weighted tensor nuclear norm (TNN) model (termed HMRMSC) to characterize real-world data. Specifically, all the similarity matrices of different views are first stacked into a third-order tensor. However, the constructed tensor may contain an additional inter-class representation since the data are usually noisy. Then, we use a technique similar to tensor principal component analysis (TPCA) to obtain a more robust similarity tensor, which is constrained by the so-called weighted TNN since the original TNN treats each singular value equally and usually considers no prior information of singular values. In addition, a high-order manifold regularized term is also added to utilize the manifold information of data. Finally, all the steps are unified into a framework, which is resolved by the augmented Lagrange multiplier (ALM) method. Experimental results on six representative datasets show that our model outperforms several state-of-the-art counterparts.

Developing Tensor-Based Common and Special Feature Analysis for Comprehensive Monitoring of Complex Batch Processes

To overcome the limitation of unfolding-based methods and handle the multiple data set and limited data problems in the complex processes, such as multigrade batch processes, a novel tensor-based common and special feature extraction method and a comprehensive monitoring framework are proposed. In the proposed method, the uneven-length three-dimensional data are directly analyzed by the comprehensive tensor-based method without unfolding. To handle the multiple data set modeling problem, the tensor-based common feature extraction methods are first proposed to obtain the common features shared among different grades. The special features are sequentially determined by conducting tensor principal component analysis (PCA) on the residuals of each grade. The data are thus divided into common, special, and residual subspaces. Three monitoring statistics are established respectively in each subspace for online fault detection. The merits and effectiveness of the proposed method are demonstrated by an injection molding process with both even-length and uneven-length data in comparison with traditional methods.

Random Tensor Theory for Tensor Decomposition

We propose a new framework for tensor decomposition based on trace invariants, which are particular cases of tensor networks. In general, tensor networks are diagrams/graphs that specify a way to "multiply" a collection of tensors together to produce another tensor, matrix or scalar. The particularity of trace invariants is that the operation of multiplying copies of a certain input tensor that produces a scalar obeys specific symmetry constraints. In other words, the scalar resulting from this multiplication is invariant under some specific transformations of the involved tensor. We focus our study on the O(N)-invariant graphs, i.e. invariant under orthogonal transformations of the input tensor. The proposed approach is novel and versatile since it allows to address different theoretical and practical aspects of both CANDECOMP/PARAFAC (CP) and Tucker decomposition models. In particular we obtain several results: (i) we generalize the computational limit of Tensor PCA (a rank-one tensor decomposition) to the case of a tensor with axes of different dimensions (ii) we introduce new algorithms for both decomposition models (iii) we obtain theoretical guarantees for these algorithms and (iv) we show improvements with respect to state of the art on synthetic and real data which also highlights a promising potential for practical applications.

Generalized Low-Rank Plus Sparse Tensor Estimation by Fast Riemannian Optimization

We investigate a generalized framework to estimate a latent low-rank plus sparse tensor, where the low-rank tensor often captures the multi-way principal components and the sparse tensor accounts for potential model mis-specifications or heterogeneous signals that are unexplainable by the low-rank part. The framework flexibly covers both linear and generalized linear models, and can easily handle continuous or categorical variables. We propose a fast algorithm by integrating the Riemannian gradient descent and a novel gradient pruning procedure. Under suitable conditions, the algorithm converges linearly and can simultaneously estimate both the low-rank and sparse tensors. The statistical error bounds of final estimates are established in terms of the gradient of loss function. The error bounds are generally sharp under specific statistical models, for example, the sub-Gaussian robust PCA and Bernoulli tensor model. Moreover, our method achieves nontrivial error bounds for heavy-tailed tensor PCA whenever the noise has a finite 2 + ε moment. We apply our method to analyze the international trade flow dataset and the statistician hypergraph coauthorship network, both yielding new and interesting findings. Supplementary materials for this article are available online.

Inference for low-rank tensors—no need to debias

In this paper, we consider the statistical inference for several low-rank tensor models. Specifically, in the Tucker low-rank tensor PCA or regression model, provided with any estimates achieving some attainable error rate, we develop the data-driven confidence regions for the singular subspace of the parameter tensor based on the asymptotic distribution of an updated estimate by two-iteration alternating minimization. The asymptotic distributions are established under some essential conditions on the signal-to-noise ratio (in PCA model) or sample size (in regression model). If the parameter tensor is further orthogonally decomposable, we develop the methods and nonasymptotic theory for inference on each individual singular vector. For the rank-one tensor PCA model, we establish the asymptotic distribution for general linear forms of principal components and confidence interval for each entry of the parameter tensor. Finally, numerical simulations are presented to corroborate our theoretical discoveries. In all of these models, we observe that different from many matrix/vector settings in existing work, debiasing is not required to establish the asymptotic distribution of estimates or to make statistical inference on low-rank tensors. In fact, due to the widely observed statistical-computational-gap for low-rank tensor estimation, one usually requires stronger conditions than the statistical (or information-theoretic) limit to ensure the computationally feasible estimation is achievable. Surprisingly, such conditions “incidentally” render a feasible low-rank tensor inference without debiasing.

New Robust Tensor PCA via Affine Transformations and L 2,1 Norms for Exact Tubal Low-Rank Recovery from Highly Corrupted and Correlated Images in Signal Processing

In this latest work, the Newly Modified Robust Tensor Principal Component Analysis (New RTPCA) using affine transformation and L 2,1 norms is proposed to remove the outliers and heavy sparse noises in signal processing. This process is done by decomposing the original data matrix as the low-rank heavy sparse noises. The determination of the potential variables is casted as constrained convex optimization problem, and the Alternating Direction Method of Multipliers (ADMM) method is considered to reduce the computational loads in an iterative manner. The simulation results validate the effectiveness of the new method as compared with that of the state-of-the-art methods.

An optimal statistical and computational framework for generalized tensor estimation

This paper describes a flexible framework for generalized low-rank tensor estimation problems that includes many important instances arising from applications in computational imaging, genomics, and network analysis. The proposed estimator consists of finding a low-rank tensor fit to the data under generalized parametric models. To overcome the difficulty of nonconvexity in these problems, we introduce a unified approach of projected gradient descent that adapts to the underlying low-rank structure. Under mild conditions on the loss function, we establish both an upper bound on statistical error and the linear rate of computational convergence through a general deterministic analysis. Then we further consider a suite of generalized tensor estimation problems, including sub-Gaussian tensor PCA, tensor regression, and Poisson and binomial tensor PCA. We prove that the proposed algorithm achieves the minimax optimal rate of convergence in estimation error. Finally, we demonstrate the superiority of the proposed framework via extensive experiments on both simulated and real data.

On the local stability of semidefinite relaxations

We consider a parametric family of quadratically constrained quadratic programs and their associated semidefinite programming (SDP) relaxations. Given a nominal value of the parameter at which the SDP relaxation is exact, we study conditions (and quantitative bounds) under which the relaxation will continue to be exact as the parameter moves in a neighborhood around the nominal value. Our framework captures a wide array of statistical estimation problems including tensor principal component analysis, rotation synchronization, orthogonal Procrustes, camera triangulation and resectioning, essential matrix estimation, system identification, and approximate GCD. Our results can also be used to analyze the stability of SOS relaxations of general polynomial optimization problems.

Accelerating the 3D T1ρ mapping of cartilage using a signal-compensated robust tensor principal component analysis model.

Magnetic resonance (MR) quantitative T1ρ imaging has been increasingly used to detect the early stages of osteoarthritis. The small volume and curved surface of articular cartilage necessitate imaging with high in-plane resolution and thin slices for accurate T1ρ measurement. Compared with 2D T1ρ mapping, 3D T1ρ mapping is free from artifacts caused by slice cross-talk and has a thinner slice thickness and full volume coverage. However, this technique needs to acquire multiple T1ρ-weighted images with different spin-lock times, which results in a very long scan duration. It is highly expected that the scan time can be reduced in 3D T1ρ mapping without compromising the T1ρ quantification accuracy and precision. To accelerate the acquisition of 3D T1ρ mapping without compromising the T1ρ quantification accuracy and precision, a signal-compensated robust tensor principal component analysis method was proposed in this paper. The 3D T1ρ-weighted images compensated at different spin-lock times were decomposed as a low-rank high-order tensor plus a sparse component. Poisson-disk random undersampling patterns were applied to k-space data in the phase- and partition-encoding directions in both retrospective and prospective experiments. Five volunteers were involved in this study. The fully sampled k-space data acquired from 3 volunteers were retrospectively undersampled at R=5.2, 7.7, and 9.7, respectively. Reference values were obtained from the fully sampled data. Prospectively undersampled data for R=5 and R=7 were acquired from 2 volunteers. Bland-Altman analyses were used to assess the agreement between the accelerated and reference T1ρ measurements. The reconstruction performance was evaluated using the normalized root mean square error and the median of the normalized absolute deviation (MNAD) of the reconstructed T1ρ-weighted images and the corresponding T1ρ maps. T1ρ parameter maps were successfully estimated from T1ρ-weighted images reconstructed using the proposed method for all accelerations. The accelerated T1ρ measurements and reference values were in good agreement for R=5.2 (T1ρ: 40.4±1.4 ms), R=7.7 (T1ρ: 40.4±2.1 ms), and R=9.7 (T1ρ: 40.9±2.2 ms) in the Bland-Altman analyses. The T1ρ parameter maps reconstructed from the prospectively undersampled data also showed promising image quality using the proposed method. The proposed method achieves the 3D T1ρ mapping of in vivo knee cartilage in eight minutes using a signal-compensated robust tensor principal component analysis method in image reconstruction.

Data Reconstruction in Heterogeneous Environmental Wireless Sensor Networks Using Robust Tensor Principal Component Analysis

The major challenges during the data acquisition process in an environmental wireless sensor network (EWSN) architecture are the presence of outliers and missing data. Outliers and missing data are ubiquitous in EWSN due to sensor failures, external noise, power dwindling, communication failures etc. Robust tensor principal component analysis (RTPCA) decomposes a noisy data tensor into a low-rank tensor and a sparse tensor, which can be exploited for the data recovery in EWSNs, where the low-rank component represents the intrinsic data tensor and the sparse component represents the gross outlier tensor. In this paper, a novel probabilistic outlier modelling scheme using multivariate Chebyshev's inequality hypothesis is proposed, which maps the sample population and the associated magnitudes of outliers with the spatio-temporal correlations in the acquired data. The inherent spatio-temporal and multi-attribute correlations in the EWSN data are established using singular value methods. A tensor nuclear norm (TNN) which extracts more temporal and multi-attribute correlations in the sensory data through block circulant matricization is used as the minimization function in RTPCA. In the forest surveillance scenario, usage of RTPCA results in a reconstruction accuracy of approximately 90% for a dataset with a high missing ratio of 0.95 and the outliers having largest possible magnitudes considered. For oceanographic data, in which the variance is lower, the simulations show similar performance for different outlier contaminations, although RPTCA performs better than other matrix and tensor data completion methods in terms of reconstruction accuracy.